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Sir: 



Applicant herewith files this brief on appeal under 37 CFR 41.37, thereby perfecting the 
notice of appeal which was originally filed on October 14, 2008. 

(1) Real Party in Interest 

This application is assigned of record to the University of Southern California who is 
hence the real party in interest. 

(2) Related Appeals and Interferences 

There are no known related appeals or interferences. 

(3) Status of Claims 

Claims 1-24, 26-28, 32, 34-36, and 40 were previously cancelled. Claims 25, 29-31, 33, 
37-39, and 41-50 are pending, with claims 29, 37, and 45 being independent. All of the pending 
claims 25, 29-31, 33, 37-39, and 41-50 stand rejected and are appealed herein. 

(4) Status of Amendments 

Claims 45-50 were amended in the Response filed on September 15, 2008, subsequent to 
the final rejection. This amendment after final has not been entered. 
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(5) Summary of Claimed Subject Matter 

The present claims define systems and techniques relating to virtual environments in 
which a model of a real environment (e.g., a battlefield) can be generated in a computer system 
and real-time video imagery can be projected onto the computer model for use in visualizing the 
real world environment. Such a system can be used in different virtual environment applications, 
"including in engineering, mission planning, training simulations, entertainment, tactical 
planning, and military operations in battlefield environments." 1 

Independent claim 29 covers a method comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 2 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 3 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 4 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; 5 and 



1 See Specification at page 6, lines 11-16. 

2 See e.g., Specification at page 7, line 21, to page 8, line 19, ref. nos. 110, 140 and 160 in FIG. 1; at page 11, lines 
9-13, ref. no. 300 in FIG. 3; and at page 1 1, line 21, to page 24, line 14, ref. nos. 500-550 in FIG. 5. 

3 See e.g., Specification at page 11, lines 13-15, ref. no. 310 in FIG. 3; and at page 36, line 15, to page 37, line 16, 
ref. no. 1510 in FIG. 15. 

4 See e.g., Specification at page 39, line 17, to page 41, line 23, ref. nos. 1600-1680 in FIG. 16. 

5 See e.g., Specification at page 11, lines 16-18, ref. no. 320 in FIG. 3. 
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visualizing the three dimensional model with the projected real-time video imagery; 6 
wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object. 7 

Independent claim 37 covers an augmented virtual environment system comprising: 
an object detection and tracking component that identifies in real time a region in motion 
with respect to a background image in real-time video imagery information from at least one 
image sensor having associated position and orientation information with respect to a three 
dimensional model of a three dimensional environment, 8 the three dimensional model generated 
from range sensor information representing a height field for the three dimensional 
environment, 9 the background image comprising a single distribution background dynamically 
modeled from a time average of the real-time video imagery information, 10 and places a surface 
that corresponds to the moving region with respect to the three dimensional model, wherein the 
object detection and tracking component places the surface by performing operations comprising 
casting a ray from an optical center, corresponding to the real-time video imagery information, to 
a bottom point of the moving region in an image plane in the three dimensional model, and 
determining a position, an orientation and a size of the surface based on the ray, a ground plane 
in the three dimensional model, and the moving region; 1 1 



6 See e.g., Specification at page 11, lines 18-20, ref. no. 330 in FIG. 3. 

7 See e.g., Specification at page 36, line 15, to page 39, line 16, ref. nos. 1520-1550 in FIG. 15. 

8 See e.g., Specification at page 5, line 16, to page 6, line 10, at page 8, line 20, to page 10, line 4, at page 24, line 
15, to page 29, line 9, ref. nos. 120, 130, 150 and 170 in FIG. 1, ref. no. 200 in FIG. 2; and at page 35, line 6, to page 
36, line 14, ref. no. 1410 in FIG. 14. 

9 See e.g., Specification at page 7, line 21, to page 8, line 19, ref. nos. 1 10, 140 and 160 in FIG. 1; at page 11, lines 
9-13, ref. no. 300 in FIG. 3; and at page 1 1, line 21, to page 24, line 14, ref. nos. 500-550 in FIG. 5. 

10 See e.g., Specification at page 36, line 15, to page 37, line 16, ref. no. 1510 in FIG. 15. 

11 See e.g., Specification at page 39, line 17, to page 41, line 23, ref. nos. 1600-1680 in FIG. 16. 
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a dynamic fusion imagery projection component that projects the real-time video imagery 
information onto the three dimensional model, including the surface, based on the position and 
orientation information; 12 and 

a visualization sub-system that visualizes the three dimensional model with the projected 
real-time video imagery; 13 

wherein the object detection and tracking component identifies the moving region by 
performing operations comprising subtracting the background image from the real-time video 
imagery information, identifying a foreground object in the subtracted real-time video imagery 
information, validating the foreground object by correlation matching between identified objects 
in neighboring image frames, and outputting the validated foreground object. 14 

Independent claim 45 covers a machine-readable medium embodying information 
indicative of instructions for causing one or more machines to perform operations comprising: 

obtairiing a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 15 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 16 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 



12 See e.g., Specification at page 6, line 7, to page 7, line 20, at page 10, lines 5-14, at page 29, line 10, to page 30, 
line 15, ref. no. 180 in FIG. 11; and at page 35, line 22, to page 36, line 14, ref. no. 1420 in FIG. 14. 

13 See e.g., Specification at page 6, line 7, to page 7, line 20, atpage 11, lines 1-8, ref. no. 190 in FIG. 1. 

14 See e.g., Specification at page 36, line 15, to page 39, line 16, ref. nos. 1520-1550 in FIG. 15. 

15 See e.g., Specification at page 7, line 21, to page 8, line 19, ref. nos. 110, 140 and 160 in FIG. 1; atpage 11, lines 
9-13, ref. no. 300 in FIG. 3; and at page 1 1, line 21, to page 24, line 14, ref. nos. 500-550 in FIG. 5. 

16 See e.g., Specification at page 11, lines 13-15, ref. no. 310 in FIG. 3; and at page 36, line 15, to page 37, line 16, 
ref.no. 1510 in FIG. 15. 
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the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 17 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; 18 and 

visualizing the three dimensional model with the projected real-time video imagery; 19 
wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object. 20 

(6) Grounds of Rejection to be Reviewed on Appeal 

I. Grounds of Rejection I 

Claims 25, 29-31, 33, 37-39, and 41-50 stand rejected under 35 U.S.C. § 103(a) as 
allegedly being unpatentable over Sawhney, H. E., et al, "Video Flashlights - Real Time 
Rendering of Multiple Videos for Immersive Model Visualization" (hereinafter "Sawhney"). 

II. Grounds of Rejection II 

Claims 45-50 stand rejected under 35 U.S.C. §112, second paragraph, as allegedly being 
indefinite. 



17 See e.g., Specification at page 39, line 17, to page 41, line 23, ref. nos. 1600-1680 in FIG. 16. 

18 See e.g., Specification at page 11, lines 16-18, ref. no. 320 in FIG. 3. 

19 See e.g., Specification at page 1 1, lines 18-20, ref. no. 330 in FIG. 3. 

20 See e.g., Specification at page 36, line 15, to page 39, line 16, ref. nos. 1520-1550 in FIG. 15. 
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(7) Argument 
I. Rejection Under 35 U.S.C. §103 

A. Claims 25, 29-31, 33, 37-39, and 41-50: 

The Sawhney reference describes an immersive model-based video visualization system 
that "provides an augmented reality solution for video surveillance and monitoring 
applications." 21 Sawhney describes a system that is similar to the presently claimed subject 
matter, but which differs in many significant respects. Thus, Sawhney fails to teach or suggest 
the claimed invention. 

First, the Office has asserted that Sawhney teaches "placing a surface that corresponds to 
the moving region in the three dimensional model" and "projecting the real-time video imagery 
information onto the three dimensional model, including the surface , based on the position and 
orientation information", on the basis that "fig. 10 illustrates moving objects". 22 However, 
Figure 10 in Sawhney shows moving object detection, not placing a surface that corresponds to 
the moving region in the three dimensional model and projecting real-time video imagery 
information onto the three dimensional model, including the surface . 23 To the contrary, moving 
objects are represented in the three dimensional model using "blobs or icons." 24 Sawhney does 
state that three dimensional models of the moving object can be "rendered along with the site 
model" 25 , but Sawhney does not state that the video information is projected onto such models. 



21 See Sawhney at page 157. 

22 See 1 1-13-2007 Office Action at page 3. 

23 See Sawhney at Figure 10 and section 5, at pages 164, 165 and 167. 

24 See Sawhney at Figures 1 1 and 12, and section 5, at pages 165, 167 and 168. 

25 See Sawhney at section 2, page 158. 
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Moreover, when this 3D modeling of dynamic objects is described in Sawhney, all that is 
presented is 3D icons with no video information projected thereon . 26 

In response to this point, the Office has further referred to Figures 8d and 9 of Sawnhey 
stating that 27 : 

In response to Applicant's argument (on page 9 in first paragraph) that the reference 
Sawhney does not state the video information is projected on to such models, Contrarily, 
Sawhney teaches clearly in fig. 8d, see model red lines. On pages 9-10, Applicant argues that 
Sawhney does not describe, placing a surface that corresponds to a moving region in a three 
dimensional model, Examiner believes Sawhney e.g., in fig. 9 illustrates a moving region (a 
moving aerial) in a 3-D model e.g., using model lines in fig. 8d. 

However, Figure 8d in Sawhney shows model lines being projected onto the image using an 
estimated pose of the camera, not projecting real-time video imagery information onto a three 
dimensional model including a surface placed into the model to correspond to a moving region 
identified in the real-time video imagery information. 

Further, the reference to Figure 9 in Sawhney is misplaced. Figure 9 in Sawhney shows 
the rendering of two frames of a video captured from a moving aerial autonomous helicopter. 
The suggestion by the Office that the "moving aerial", which is the helicopter having a camera 
that captures the video imagery , can somehow be equated with the moving region identified 
within the video imagery itself completely defies common sense, and contradicts the other 
stated bases for rejecting the claims. The Advisory Action dated 10-2-2008 fails to clarify the 
logic behind the reference to the "moving aerial" in Sawhney or how this can serve as a basis to 

26 See Sawhney at Figure 12, and section 5, at pages 165 and 168. 

27 See 6-13-2008 Office Action at page 2. 
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reject the claims. Thus, Sawhney fails to teach or suggest projecting real-time video imagery 
information onto a surface that has been placed into a three dimensional model of a three 
dimensional environment in order to correspond to a region in motion identified within that very 
same real-time video imagery information, and the rejection should be overturned for at least this 
reason. 

Second, claim 29 further recites, "wherein placing the surface comprises casting a ray 
from an optical center, corresponding to the real-time video imagery information, to a bottom 
point of the moving region in an image plane in the three dimensional model, and determining a 
position, an orientation and a size of the surface based on the ray, a ground plane in the three 
dimensional model, and the moving region." In the rejection of this subject matter, the Office 
has stated, "Examiner's note: casting rays are: cast and traced in groups based on some geometric 
constraints, and each ray is traced separately, so that every point (usually a pixel) on the display 
is traced by one ray." 28 However, the presently claimed ray casting has nothing to do with 
tracing pixel points on a display, as is evident in view of the plain meaning of the claim language 
and the Specification. 29 Rather, this claimed subject matter relates to casting a vector in the three 
dimensional model. The Office has also referred to Figures 5 and 6 of Sawhney, but these 
portions of Sawhney simply show video rendering on the site model alone, and do not relate to or 
describe placing a surface that corresponds to a moving region in a three dimensional model. 

Furthermore, with respect to "determining a position, an orientation and a size of the 
surface based on the ray, a ground plane in the three dimensional model, and the moving region", 



See 6-13-2008 Office Action at page 5. 

See e.g., Specification at page 39, line 17, to page 41, line 23, and at FIG. 16. 
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the Office states, "fig. 10 illustrates a 3D model, and a moving region." 30 With all due respect to 
the Office, Figure 10 and its corresponding description discloses moving object detection, but 
says nothing about determining a position, an orientation and a size of a surface that is placed in 
a three dimensional model, let alone "casting a ray from an optical center , corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane 
in the three dimensional model." In fact, the only technique described in Sawhney for modeling 
a moving object is to use "stereo depth information" from "stereo cameras", 31 which does not in 
any way teach or suggest "placing a surface that corresponds to the moving region in the three 
dimensional model, wherein placing the surface comprises casting a ray from an optical center , 
corresponding to the real-time video imagery information, to a bottom point of the moving 
region in an image plane in the three dimensional model, and determining a position, an 
orientation and a size of the surface based on the ray , a ground plane in the three dimensional 
model, and the moving region ." 

Thus, Sawhney fails to teach or suggest this subject matter, as claimed, and the rejection 
of claim 29 should be overturned for at least this additional reason. Similar arguments are 
applicable to independent claims 37 and 45. Thus, for all of the above reasons, Ground of 
Rejection I should be overturned. 

B. Claims 41, 43, and 49: 

Claims 41, 43 and 49 each recite, "wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time video imagery information using a 

30 See 6-13-2008 Office Action at page 5. 

31 See Sawhney at sections 2 and 5, at pages 158 and 165. 
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histogram-based threshold and a noise filter." The cited portion of Sawhney clearly describes 
the use of a threshold applied to a difference image between a current image and a median of a 
stack of images. 32 Sawhney does not indicate that the threshold is histogram-based or that a 
noise filter is used. In response, the Office has referred to page 164 and Sawhney's teaching that 
"the absolute value at each point can then be computed, and the result can be thresholded in 
order to highlight intensity or feature differences between the current video image and the 
reference image[.]" 33 But the Office has failed to explain how this teaching of Sawhney is 
considered to teach the use of a histogram-based threshold. Since Sawhney fails to teach or 
suggest this feature of the claim, the rejection should be overturned for at least this reason. 

Furthermore, the Office's interpretation that "a noise filter" may be considered "as a 
virtual camera view in fig. 3" 34 is without rational basis. In response to this point, the Office has 
referred to page 160 and Figure 3 of Sawhney, asserting that such "clearly illustrates final 
rendered composite that has been filtered the video pixels." 35 But this does not address the 
claimed subject matter, which requires the use of a noise filter in the process of identifying a 
foreground object . Sawhney never describes such a use of a noise filter. 36 Thus, the rejection 
should be overturned for at least this additional reason. 

In view of these further arguments, Ground of Rejection I should be overturned with 
respect to claims 41, 43 and 49. 



32 See Sawhney at Figure 10 and section 5, at page 164 and 167. 

33 See 6-13-2008 Office Action at pages 2-3. 

34 See 1 1-13-2007 Office Action at page 4. 

35 See 6-13-2008 Office Action at page 3. 

36 See Sawney in section 5, at pages 164-165. 
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C. Claims 42, 44, and 50: 

Claims 42, 44 and 50 each recite, "wherein identifying a region in motion in real time 
further comprises estimating the background image by modeling the background image as a 
temporal pixel average of five recent image frames in the real-time video imagery information." 
With respect to this subject matter, the Office "takes an official notice for an average of five 
recent frames, because Sawhney on page 161 under 'A pseudo-code for the rendering algorithm 
..' teaches as one of the parameters that a user may be indicated a number for number of frames 
'frame number', and that number may be 5 image frames." 37 This official notice was explicitly 
challenged. The "Frame Number" here is a parameter to the "Update VideoContent" procedure, 
which clearly relates to retrieving the most recent available video data from a camera specified 
by "Video Source". Contrary to the Office's assertion, nothing in Sawhney suggests that this 
procedure is part of the process of moving object detection or that the "Frame Number" may be 
an indication to use five recent image frames to generate a temporal pixel average to model the 
background image. 

Furthermore, taking official notice regarding this claimed subject matter is inappropriate 

since the Office has previously contended that "applicant has established criticality for the use of 

five frames in the averaging technique" and "optimizing such a number of frames would require 

undue experimentation under In re Wands, since applicant points out that five frames has a 

specific benefit, as per the specification." 38 In response to these points, the Office states 39 : 

(e.g., on page 164 at right col. teaches the reference background image needs to 
be constantly updated during the day to reflect changing ambient illumination. 

37 See 1 1-13-2007 Office Action at page 4. 

38 See 1 1-28-2005 Office Action at page 6. 

39 See 6-12-2008 Office Action at page 3. 
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Alternatively only moving objects may be detected by comparing current image 
with an image taken a few seconds before. The 2D moving object detection can 
be done in real time, also see page 162 left column at third paragraph). On the 
other hand, comparing fig. 15 of the current invention with fig. 10 of the 
reference: in fig. 15 steps 1500, 1510, 1520, and 1530 illustrate what is shown the 
left side of fig. 10, and fig. 10 illustrates the validated foreground object in step 
1550 in the middle of fig. 10. 
None of these points by the Office actually address the argument regarding the Office's official 

notice. Since the Office has admitted that Sawhney does not teach "estimating the background 

image by modeling the background image as a temporal pixel average of five recent image 

frames in the real-time video imagery information", and since the Office's taking of official 

notice cannot be reconciled with the Office's prior assertions that "applicant has established 

criticality for the use of five frames in the averaging technique" and "optimizing such a number 

of frames would require undue experimentation under In re Wands, since applicant points out 

that five frames has a specific benefit, as per the specification", the rejection should be 

overturned. 

In view of these further arguments, Ground of Rejection I should be overturned with 
respect to claims 42, 44 and 50. 

II. Rejection Under 35 U.S.C. §112 

Claims 45-50 were rejected under 35 U.S.C. §112, second paragraph, as allegedly being 
indefinite. The Office stated that, "the machine-readable medium is not well established whether 
is a camera, a computer or an optical device. The limitation of 'medium' is not explicitly 
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defined whether is a storage device, or a signal, etc." 40 An amendment to claims 45-50 was 
presented in the Response filed on September 15, 2008, to obviate the rejection by replacing 
"medium" with "storage device" (without conceding the propriety of the rejection). The 
Advisory Action dated 10-2-2008 did not specify whether or not the amendment had been 
entered, but during a telephone call on January 13, 2009, Examiner Amini indicated that the 
amendment had not been entered. When asked why, Examiner Amini indicated that he should 
have objected to the specification, rather than reject the claims. In light of this, and in light of 
the facts that (1) the plain meaning of the phrase "machine-readable medium" is well established 
in view of the original application as filed and the knowledge of those of ordinary skill in the art, 
and (2) the Specification never suggests that a "medium" can be a "signal", Ground of Rejection 
II should be overturned. 

Please apply the appeal brief fee, the extension of time fee, and any other necessary 
charges or credits to Deposit Account No. 06-1050. 

Respectfully submitted, 




William E. Hunter 
Reg. No. 47,671 



Fish & Richardson P.C. 
PTO Customer No. 20985 
Telephone: (858) 678-5070 
Facsimile: (877) 769-7945 



10879985.doc 



See 6-13-2008 Office Action at page 9. 
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Appendix of Claims 



25. The method of claim 29, wherein the surface comprises a two dimensional surface. 

29. A method comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; 

wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object. 

30. The method of claim 29, further comprising tracking the position and orientation 
information of the at least one image sensor in the environment with respect to the three 
dimensional model in real-time. 
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3 1 . The method of claim 30, wherein obtaining a three dimensional model of a three 
dimensional environment comprises generating the three dimensional model of the three 
dimensional environment. 

33. The system of claim 37, wherein the surface comprises a two dimensional surface. 

37. An augmented virtual environment system comprising: 

an object detection and tracking component that identifies in real time a region in motion 
with respect to a background image in real-time video imagery information from at least one 
image sensor having associated position and orientation information with respect to a three 
dimensional model of a three dimensional environment, the three dimensional model generated 
from range sensor information representing a height field for the three dimensional environment, 
the background image comprising a single distribution background dynamically modeled from a 
time average of the real-time video imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model, wherein the object detection and 
tracking component places the surface by performing operations comprising casting a ray from 
an optical center, corresponding to the real-time video imagery information, to a bottom point of 
the moving region in an image plane in the three dimensional model, and determining a position, 
an orientation and a size of the surface based on the ray, a ground plane in the three dimensional 
model, and the moving region; 

a dynamic fusion imagery projection component that projects the real-time video imagery 
information onto the three dimensional model, including the surface, based on the position and 
orientation information; and 

a visualization sub-system that visualizes the three dimensional model with the projected 
real-time video imagery; 

wherein the object detection and tracking component identifies the moving region by 
performing operations comprising subtracting the background image from the real-time video 
imagery information, identifying a foreground object in the subtracted real-time video imagery 
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information, validating the foreground object by correlation matching between identified objects 
in neighboring image frames, and outputting the validated foreground object. 

38. The system of claim 37, further comprising a tracking sensor system that integrates 
visual input, global navigational satellite system receiver input, and inertial orientation sensor 
input to obtain the position and the orientation information associated with the at least one image 
sensor in real time in conjunction with the real-time video imagery. 

39. The system of claim 38, further comprising a model construction component that 
generates the three dimensional model of the three dimensional environment. 

41. The method of claim 29, wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time video imagery information using a 
histogram-based threshold and a noise filter. 

42. The method of claim 41, wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the background image as a temporal 
pixel average of five recent image frames in the real-time video imagery information. 

43. The system of claim 37, wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time video imagery information using a 
histogram-based threshold and a noise filter. 

44. The system of claim 43, wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the background image as a temporal 
pixel average of five recent image frames in the real-time video imagery information. 
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45. A machine-readable medium embodying information indicative of instructions for 
causing one or more machines to perform operations comprising: 

obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the 
three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- 
time video imagery information from at least one image sensor having associated position and 
orientation information with respect to the three dimensional model, the background image 
comprising a single distribution background dynamically modeled from a time average of the 
real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, 
wherein placing the surface comprises casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model, and determining a position, an orientation and a size of the surface 
based on the ray, a ground plane in the three dimensional model, and the moving region; 

projecting the real-time video imagery information onto the three dimensional model, 
including the surface, based on the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; 

wherein identifying a region in motion in real time comprises subtracting the background 
image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the foreground object by correlation 
matching between identified objects in neighboring image frames, and outputting the validated 
foreground object. 

46. The machine-readable medium of claim 45, wherein the surface comprises a two 
dimensional surface. 

47. The machine-readable medium of claim 45, further comprising tracking the position 
and orientation information of the at least one image sensor in the environment with respect to 
the three dimensional model in real-time. 
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48. The machine-readable medium of claim 47, wherein obtaining a three dimensional 
model of a three dimensional environment comprises generating the three dimensional model of 
the three dimensional environment. 

49. The machine-readable medium of claim 45, wherein identifying a foreground object 
comprises identifying the foreground object in the subtracted real-time video imagery 
information using a histogram-based threshold and a noise filter. 

50. The machine-readable medium of claim 45, wherein identifying a region in motion in 
real time further comprises estimating the background image by modeling the background image 
as a temporal pixel average of five recent image frames in the real-time video imagery 
information. 
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